Skip to content

Update Spock Tests to run with STRICT_TEST mode by default#11032

Open
mhlidd wants to merge 18 commits intomasterfrom
mhlidd/run_spock_with_strict
Open

Update Spock Tests to run with STRICT_TEST mode by default#11032
mhlidd wants to merge 18 commits intomasterfrom
mhlidd/run_spock_with_strict

Conversation

@mhlidd
Copy link
Copy Markdown
Contributor

@mhlidd mhlidd commented Apr 1, 2026 •
edited
Loading

What Does This Do

This PR updates Spock tests to run with STRICT_TEST by default. This means that all Environment Variables that are queried but not documented will fail the test. We fail the test by throwing an error when querying such environment variable, but if the test has automatic error handling we also throw an error during test cleanup to log the missing configurations from supported-configurations.json.

If a test is using fake configurations, the test itself can override the strictness mode with TEST to bypass this mechanism. Note that this is being done w/ many Instrumentation tests as they create fake decorators that register fake configs.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

mhlidd added 14 commits April 1, 2026 15:38
@mhlidd
updating config helper to have descriptive exception in strict mode
4c8153c
@mhlidd
enabling ConfigHelper strict mode by default for spock tests
f29cdf5
@mhlidd
updating test-utils to include config-utils dependency at runtime
1a30830
@mhlidd
overriding strictness for tests using fake env vars
da699c2
@mhlidd
test adding real env vars
650e7d5
@mhlidd
updating tests using fake env vars and adding more env vars
14e0405
@mhlidd
overriding test mode for elasticsearch and opensearch
b998bec
@mhlidd
adding final integration configs and overriding fake config tests
3e2ccfe
@mhlidd
updating strictness in configurePreAgent
92e6d26
@mhlidd
update config helper to guarantee throw an error that won't be caught
3582e00
@mhlidd
configuring more fake config tests
e7b23e9
@mhlidd
adding final configurations
b618154
@mhlidd
Merge branch 'master' into mhlidd/run_spock_with_strict
84dc914
@mhlidd
revert supported-configurations
9d52790
@mhlidd mhlidd changed the base branch from master to mhlidd/update_strictness_policy April 2, 2026 21:09
@mhlidd mhlidd changed the base branch from mhlidd/update_strictness_policy to master April 2, 2026 21:10
@mhlidd mhlidd mentioned this pull request Apr 5, 2026
Merged
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 5, 2026 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775589541 1775596802
git_commit_sha aa7c70f e947d7d
release_version 1.61.0-SNAPSHOT~aa7c70f2e7 1.61.0-SNAPSHOT~e947d7dd42
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775598604 1775598604
ci_job_id 1575385047 1575385047
ci_pipeline_id 106526818 106526818
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-kpret59z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-kpret59z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 10 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059709
Total [baseline] (11.176 s) : 0, 11176075
Agent [candidate] (1.068 s) : 0, 1068442
Total [candidate] (11.085 s) : 0, 11084825
section appsec
Agent [baseline] (1.255 s) : 0, 1255268
Total [baseline] (11.18 s) : 0, 11180141
Agent [candidate] (1.262 s) : 0, 1261992
Total [candidate] (11.304 s) : 0, 11303561
section iast
Agent [baseline] (1.228 s) : 0, 1227928
Total [baseline] (11.354 s) : 0, 11354233
Agent [candidate] (1.238 s) : 0, 1238044
Total [candidate] (11.368 s) : 0, 11367604
section profiling
Agent [baseline] (1.191 s) : 0, 1191137
Total [baseline] (10.969 s) : 0, 10969036
Agent [candidate] (1.188 s) : 0, 1187598
Total [candidate] (11.265 s) : 0, 11264769
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.06 s -
Agent appsec 1.255 s 195.56 ms (18.5%)
Agent iast 1.228 s 168.219 ms (15.9%)
Agent profiling 1.191 s 131.428 ms (12.4%)
Total tracing 11.176 s -
Total appsec 11.18 s 4.066 ms (0.0%)
Total iast 11.354 s 178.158 ms (1.6%)
Total profiling 10.969 s -207.039 ms (-1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.068 s -
Agent appsec 1.262 s 193.55 ms (18.1%)
Agent iast 1.238 s 169.602 ms (15.9%)
Agent profiling 1.188 s 119.157 ms (11.2%)
Total tracing 11.085 s -
Total appsec 11.304 s 218.737 ms (2.0%)
Total iast 11.368 s 282.779 ms (2.6%)
Total profiling 11.265 s 179.945 ms (1.6%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.234 ms) : 0, 1234
crashtracking [candidate] (1.247 ms) : 0, 1247
BytebuddyAgent [baseline] (633.073 ms) : 0, 633073
BytebuddyAgent [candidate] (641.287 ms) : 0, 641287
AgentMeter [baseline] (29.467 ms) : 0, 29467
AgentMeter [candidate] (29.487 ms) : 0, 29487
GlobalTracer [baseline] (249.352 ms) : 0, 249352
GlobalTracer [candidate] (250.503 ms) : 0, 250503
AppSec [baseline] (32.041 ms) : 0, 32041
AppSec [candidate] (32.171 ms) : 0, 32171
Debugger [baseline] (59.852 ms) : 0, 59852
Debugger [candidate] (60.408 ms) : 0, 60408
Remote Config [baseline] (598.094 µs) : 0, 598
Remote Config [candidate] (607.642 µs) : 0, 608
Telemetry [baseline] (8.119 ms) : 0, 8119
Telemetry [candidate] (8.17 ms) : 0, 8170
Flare Poller [baseline] (9.744 ms) : 0, 9744
Flare Poller [candidate] (8.214 ms) : 0, 8214
section appsec
crashtracking [baseline] (1.239 ms) : 0, 1239
crashtracking [candidate] (1.25 ms) : 0, 1250
BytebuddyAgent [baseline] (665.418 ms) : 0, 665418
BytebuddyAgent [candidate] (672.279 ms) : 0, 672279
AgentMeter [baseline] (12.125 ms) : 0, 12125
AgentMeter [candidate] (12.142 ms) : 0, 12142
GlobalTracer [baseline] (250.731 ms) : 0, 250731
GlobalTracer [candidate] (250.784 ms) : 0, 250784
AppSec [baseline] (185.351 ms) : 0, 185351
AppSec [candidate] (184.911 ms) : 0, 184911
Debugger [baseline] (66.197 ms) : 0, 66197
Debugger [candidate] (66.479 ms) : 0, 66479
Remote Config [baseline] (610.85 µs) : 0, 611
Remote Config [candidate] (608.946 µs) : 0, 609
Telemetry [baseline] (8.708 ms) : 0, 8708
Telemetry [candidate] (8.69 ms) : 0, 8690
Flare Poller [baseline] (3.603 ms) : 0, 3603
Flare Poller [candidate] (3.612 ms) : 0, 3612
IAST [baseline] (24.734 ms) : 0, 24734
IAST [candidate] (24.634 ms) : 0, 24634
section iast
crashtracking [baseline] (1.246 ms) : 0, 1246
crashtracking [candidate] (1.243 ms) : 0, 1243
BytebuddyAgent [baseline] (804.315 ms) : 0, 804315
BytebuddyAgent [candidate] (813.817 ms) : 0, 813817
AgentMeter [baseline] (11.384 ms) : 0, 11384
AgentMeter [candidate] (11.621 ms) : 0, 11621
GlobalTracer [baseline] (239.356 ms) : 0, 239356
GlobalTracer [candidate] (240.474 ms) : 0, 240474
AppSec [baseline] (31.854 ms) : 0, 31854
AppSec [candidate] (32.271 ms) : 0, 32271
Debugger [baseline] (61.476 ms) : 0, 61476
Debugger [candidate] (58.046 ms) : 0, 58046
Remote Config [baseline] (551.916 µs) : 0, 552
Remote Config [candidate] (1.722 ms) : 0, 1722
Telemetry [baseline] (12.212 ms) : 0, 12212
Telemetry [candidate] (12.931 ms) : 0, 12931
Flare Poller [baseline] (3.469 ms) : 0, 3469
Flare Poller [candidate] (3.471 ms) : 0, 3471
IAST [baseline] (25.751 ms) : 0, 25751
IAST [candidate] (25.94 ms) : 0, 25940
section profiling
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (696.314 ms) : 0, 696314
BytebuddyAgent [candidate] (693.51 ms) : 0, 693510
AgentMeter [baseline] (9.268 ms) : 0, 9268
AgentMeter [candidate] (9.148 ms) : 0, 9148
GlobalTracer [baseline] (208.092 ms) : 0, 208092
GlobalTracer [candidate] (207.261 ms) : 0, 207261
AppSec [baseline] (32.495 ms) : 0, 32495
AppSec [candidate] (32.609 ms) : 0, 32609
Debugger [baseline] (65.81 ms) : 0, 65810
Debugger [candidate] (65.845 ms) : 0, 65845
Remote Config [baseline] (572.416 µs) : 0, 572
Remote Config [candidate] (581.757 µs) : 0, 582
Telemetry [baseline] (7.802 ms) : 0, 7802
Telemetry [candidate] (7.882 ms) : 0, 7882
Flare Poller [baseline] (3.55 ms) : 0, 3550
Flare Poller [candidate] (3.616 ms) : 0, 3616
ProfilingAgent [baseline] (94.33 ms) : 0, 94330
ProfilingAgent [candidate] (94.485 ms) : 0, 94485
Profiling [baseline] (94.902 ms) : 0, 94902
Profiling [candidate] (95.065 ms) : 0, 95065
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062857
Total [baseline] (8.856 s) : 0, 8856366
Agent [candidate] (1.054 s) : 0, 1053739
Total [candidate] (8.867 s) : 0, 8866856
section iast
Agent [baseline] (1.229 s) : 0, 1229442
Total [baseline] (9.61 s) : 0, 9610004
Agent [candidate] (1.222 s) : 0, 1222295
Total [candidate] (9.561 s) : 0, 9561360
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.063 s -
Agent iast 1.229 s 166.585 ms (15.7%)
Total tracing 8.856 s -
Total iast 9.61 s 753.638 ms (8.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent iast 1.222 s 168.557 ms (16.0%)
Total tracing 8.867 s -
Total iast 9.561 s 694.504 ms (7.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.225 ms) : 0, 1225
BytebuddyAgent [baseline] (637.804 ms) : 0, 637804
BytebuddyAgent [candidate] (632.796 ms) : 0, 632796
AgentMeter [baseline] (29.595 ms) : 0, 29595
AgentMeter [candidate] (29.25 ms) : 0, 29250
GlobalTracer [baseline] (249.237 ms) : 0, 249237
GlobalTracer [candidate] (248.689 ms) : 0, 248689
AppSec [baseline] (31.868 ms) : 0, 31868
AppSec [candidate] (31.911 ms) : 0, 31911
Debugger [baseline] (59.252 ms) : 0, 59252
Debugger [candidate] (59.214 ms) : 0, 59214
Remote Config [baseline] (597.373 µs) : 0, 597
Remote Config [candidate] (601.7 µs) : 0, 602
Telemetry [baseline] (8.054 ms) : 0, 8054
Telemetry [candidate] (8.033 ms) : 0, 8033
Flare Poller [baseline] (8.911 ms) : 0, 8911
Flare Poller [candidate] (5.82 ms) : 0, 5820
section iast
crashtracking [baseline] (1.244 ms) : 0, 1244
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (806.18 ms) : 0, 806180
BytebuddyAgent [candidate] (800.816 ms) : 0, 800816
AgentMeter [baseline] (11.552 ms) : 0, 11552
AgentMeter [candidate] (11.376 ms) : 0, 11376
GlobalTracer [baseline] (239.414 ms) : 0, 239414
GlobalTracer [candidate] (239.02 ms) : 0, 239020
AppSec [baseline] (30.3 ms) : 0, 30300
AppSec [candidate] (33.338 ms) : 0, 33338
Debugger [baseline] (61.026 ms) : 0, 61026
Debugger [candidate] (58.258 ms) : 0, 58258
Remote Config [baseline] (1.168 ms) : 0, 1168
Remote Config [candidate] (1.11 ms) : 0, 1110
Telemetry [baseline] (12.942 ms) : 0, 12942
Telemetry [candidate] (11.559 ms) : 0, 11559
Flare Poller [baseline] (3.463 ms) : 0, 3463
Flare Poller [candidate] (3.461 ms) : 0, 3461
IAST [baseline] (25.888 ms) : 0, 25888
IAST [candidate] (25.823 ms) : 0, 25823
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775589541 1775596802
git_commit_sha aa7c70f e947d7d
release_version 1.61.0-SNAPSHOT~aa7c70f2e7 1.61.0-SNAPSHOT~e947d7dd42
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775599228 1775599228
ci_job_id 1575385048 1575385048
ci_pipeline_id 106526818 106526818
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-8dobsvqy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-8dobsvqy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 0 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast:high_load better
[-191.644µs; -101.316µs] or [-7.223%; -3.818%]		 unsure
[-509.036µs; -141.011µs] or [-6.670%; -1.848%]		 unstable
[-85.895op/s; +232.895op/s] or [-6.363%; +17.253%]		 2.507ms 7.307ms 1423.406op/s 2.653ms 7.632ms 1349.906op/s
scenario:load:insecure-bank:iast_FULL:high_load better
[-434.047µs; -265.623µs] or [-7.632%; -4.670%]		 better
[-1315.225µs; -670.892µs] or [-9.685%; -4.940%]		 unstable
[-35.439op/s; +129.564op/s] or [-4.882%; +17.849%]		 5.338ms 12.587ms 772.969op/s 5.688ms 13.580ms 725.906op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.258 ms) : 1245, 1271
.   : milestone, 1258,
iast (3.394 ms) : 3345, 3443
.   : milestone, 3394,
iast_FULL (6.377 ms) : 6311, 6444
.   : milestone, 6377,
iast_GLOBAL (3.717 ms) : 3652, 3781
.   : milestone, 3717,
profiling (2.224 ms) : 2202, 2246
.   : milestone, 2224,
tracing (1.866 ms) : 1850, 1881
.   : milestone, 1866,
section candidate
no_agent (1.246 ms) : 1234, 1258
.   : milestone, 1246,
iast (3.215 ms) : 3171, 3258
.   : milestone, 3215,
iast_FULL (5.985 ms) : 5925, 6045
.   : milestone, 5985,
iast_GLOBAL (3.672 ms) : 3608, 3736
.   : milestone, 3672,
profiling (2.179 ms) : 2158, 2199
.   : milestone, 2179,
tracing (1.945 ms) : 1928, 1963
.   : milestone, 1945,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.258 ms [1.245 ms, 1.271 ms] -
iast 3.394 ms [3.345 ms, 3.443 ms] 2.136 ms (169.8%)
iast_FULL 6.377 ms [6.311 ms, 6.444 ms] 5.119 ms (406.9%)
iast_GLOBAL 3.717 ms [3.652 ms, 3.781 ms] 2.459 ms (195.4%)
profiling 2.224 ms [2.202 ms, 2.246 ms] 966.179 µs (76.8%)
tracing 1.866 ms [1.85 ms, 1.881 ms] 607.547 µs (48.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.246 ms [1.234 ms, 1.258 ms] -
iast 3.215 ms [3.171 ms, 3.258 ms] 1.969 ms (158.0%)
iast_FULL 5.985 ms [5.925 ms, 6.045 ms] 4.739 ms (380.4%)
iast_GLOBAL 3.672 ms [3.608 ms, 3.736 ms] 2.426 ms (194.7%)
profiling 2.179 ms [2.158 ms, 2.199 ms] 932.694 µs (74.9%)
tracing 1.945 ms [1.928 ms, 1.963 ms] 699.583 µs (56.2%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.17 ms) : 17985, 18354
.   : milestone, 18170,
appsec (19.057 ms) : 18866, 19249
.   : milestone, 19057,
code_origins (18.189 ms) : 18012, 18365
.   : milestone, 18189,
iast (18.16 ms) : 17980, 18340
.   : milestone, 18160,
profiling (18.486 ms) : 18301, 18671
.   : milestone, 18486,
tracing (18.167 ms) : 17986, 18347
.   : milestone, 18167,
section candidate
no_agent (19.382 ms) : 19187, 19576
.   : milestone, 19382,
appsec (18.868 ms) : 18679, 19058
.   : milestone, 18868,
code_origins (17.865 ms) : 17688, 18042
.   : milestone, 17865,
iast (18.021 ms) : 17843, 18199
.   : milestone, 18021,
profiling (18.424 ms) : 18241, 18607
.   : milestone, 18424,
tracing (17.644 ms) : 17469, 17819
.   : milestone, 17644,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.17 ms [17.985 ms, 18.354 ms] -
appsec 19.057 ms [18.866 ms, 19.249 ms] 887.549 µs (4.9%)
code_origins 18.189 ms [18.012 ms, 18.365 ms] 19.267 µs (0.1%)
iast 18.16 ms [17.98 ms, 18.34 ms] -9.485 µs (-0.1%)
profiling 18.486 ms [18.301 ms, 18.671 ms] 316.542 µs (1.7%)
tracing 18.167 ms [17.986 ms, 18.347 ms] -2.983 µs (-0.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.382 ms [19.187 ms, 19.576 ms] -
appsec 18.868 ms [18.679 ms, 19.058 ms] -513.165 µs (-2.6%)
code_origins 17.865 ms [17.688 ms, 18.042 ms] -1.517 ms (-7.8%)
iast 18.021 ms [17.843 ms, 18.199 ms] -1.36 ms (-7.0%)
profiling 18.424 ms [18.241 ms, 18.607 ms] -957.705 µs (-4.9%)
tracing 17.644 ms [17.469 ms, 17.819 ms] -1.737 ms (-9.0%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775589541 1775596802
git_commit_sha aa7c70f e947d7d
release_version 1.61.0-SNAPSHOT~aa7c70f2e7 1.61.0-SNAPSHOT~e947d7dd42
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1775598873 1775598873
ci_job_id 1575385050 1575385050
ci_pipeline_id 106526818 106526818
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-gby18igu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-gby18igu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.562 s) : 15562000, 15562000
.   : milestone, 15562000,
appsec (14.901 s) : 14901000, 14901000
.   : milestone, 14901000,
iast (18.271 s) : 18271000, 18271000
.   : milestone, 18271000,
iast_GLOBAL (17.9 s) : 17900000, 17900000
.   : milestone, 17900000,
profiling (15.057 s) : 15057000, 15057000
.   : milestone, 15057000,
tracing (14.935 s) : 14935000, 14935000
.   : milestone, 14935000,
section candidate
no_agent (15.178 s) : 15178000, 15178000
.   : milestone, 15178000,
appsec (14.913 s) : 14913000, 14913000
.   : milestone, 14913000,
iast (18.26 s) : 18260000, 18260000
.   : milestone, 18260000,
iast_GLOBAL (18.061 s) : 18061000, 18061000
.   : milestone, 18061000,
profiling (15.115 s) : 15115000, 15115000
.   : milestone, 15115000,
tracing (14.924 s) : 14924000, 14924000
.   : milestone, 14924000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.562 s [15.562 s, 15.562 s] -
appsec 14.901 s [14.901 s, 14.901 s] -661.0 ms (-4.2%)
iast 18.271 s [18.271 s, 18.271 s] 2.709 s (17.4%)
iast_GLOBAL 17.9 s [17.9 s, 17.9 s] 2.338 s (15.0%)
profiling 15.057 s [15.057 s, 15.057 s] -505.0 ms (-3.2%)
tracing 14.935 s [14.935 s, 14.935 s] -627.0 ms (-4.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.178 s [15.178 s, 15.178 s] -
appsec 14.913 s [14.913 s, 14.913 s] -265.0 ms (-1.7%)
iast 18.26 s [18.26 s, 18.26 s] 3.082 s (20.3%)
iast_GLOBAL 18.061 s [18.061 s, 18.061 s] 2.883 s (19.0%)
profiling 15.115 s [15.115 s, 15.115 s] -63.0 ms (-0.4%)
tracing 14.924 s [14.924 s, 14.924 s] -254.0 ms (-1.7%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~e947d7dd42, baseline=1.61.0-SNAPSHOT~aa7c70f2e7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.493 ms) : 1482, 1505
.   : milestone, 1493,
appsec (3.852 ms) : 3628, 4076
.   : milestone, 3852,
iast (2.297 ms) : 2228, 2367
.   : milestone, 2297,
iast_GLOBAL (2.336 ms) : 2265, 2406
.   : milestone, 2336,
profiling (2.11 ms) : 2055, 2165
.   : milestone, 2110,
tracing (2.101 ms) : 2046, 2155
.   : milestone, 2101,
section candidate
no_agent (1.495 ms) : 1483, 1507
.   : milestone, 1495,
appsec (3.877 ms) : 3653, 4101
.   : milestone, 3877,
iast (2.287 ms) : 2217, 2356
.   : milestone, 2287,
iast_GLOBAL (2.338 ms) : 2268, 2409
.   : milestone, 2338,
profiling (2.135 ms) : 2079, 2191
.   : milestone, 2135,
tracing (2.095 ms) : 2041, 2150
.   : milestone, 2095,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.493 ms [1.482 ms, 1.505 ms] -
appsec 3.852 ms [3.628 ms, 4.076 ms] 2.358 ms (157.9%)
iast 2.297 ms [2.228 ms, 2.367 ms] 803.659 µs (53.8%)
iast_GLOBAL 2.336 ms [2.265 ms, 2.406 ms] 842.192 µs (56.4%)
profiling 2.11 ms [2.055 ms, 2.165 ms] 616.541 µs (41.3%)
tracing 2.101 ms [2.046 ms, 2.155 ms] 607.042 µs (40.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.495 ms [1.483 ms, 1.507 ms] -
appsec 3.877 ms [3.653 ms, 4.101 ms] 2.382 ms (159.3%)
iast 2.287 ms [2.217 ms, 2.356 ms] 791.622 µs (52.9%)
iast_GLOBAL 2.338 ms [2.268 ms, 2.409 ms] 843.343 µs (56.4%)
profiling 2.135 ms [2.079 ms, 2.191 ms] 639.814 µs (42.8%)
tracing 2.095 ms [2.041 ms, 2.15 ms] 600.452 µs (40.2%)

@mhlidd mhlidd added type: enhancement Enhancements and improvements comp: testing Testing comp: config Configuration labels Apr 6, 2026
@mhlidd mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 5ba6f69 to b0147c9 Compare April 6, 2026 17:00
@mhlidd mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 95b7e3f to 82a89e0 Compare April 6, 2026 18:20
@mhlidd mhlidd marked this pull request as ready for review April 6, 2026 20:51
@mhlidd mhlidd requested review from a team as code owners April 6, 2026 20:51
@vandonr
Copy link
Copy Markdown
Contributor

vandonr commented Apr 7, 2026

@codex review

@vandonr
Copy link
Copy Markdown
Contributor

vandonr commented Apr 7, 2026

ok so most of the changes are coming from the test files where we override that global config

@chatgpt-codex-connector
Copy link
Copy Markdown

chatgpt-codex-connector bot commented Apr 7, 2026

Codex Review: Didn't find any major issues. What shall we delve into next?

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mhlidd mhlidd added this pull request to the merge queue Apr 7, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts bot commented Apr 7, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-04-07 15:06:47 UTC ℹ️ Start processing command /merge

2026-04-07 15:06:52 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-04-07 16:11:56 UTC ℹ️ MergeQueue: Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.

2026-04-07 18:13:10 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 106438802 took longer than expected. The current limit for the base branch 'master' is 120 minutes.

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 7, 2026
@mhlidd mhlidd added this pull request to the merge queue Apr 7, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts bot commented Apr 7, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-04-07 18:29:22 UTC ℹ️ Start processing command /merge

2026-04-07 18:29:26 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-04-07 21:19:07 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 106484564 took longer than expected. The current limit for the base branch 'master' is 120 minutes.

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vandonr vandonr vandonr approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/asm-java

@claponcet claponcet Awaiting requested review from claponcet claponcet is a code owner automatically assigned from DataDog/asm-java

Assignees

No one assigned

Labels

comp: config Configuration comp: testing Testing type: enhancement Enhancements and improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mhlidd @vandonr